Continuous online extraction of HTTP traces from packet traces

To improve the performance of the network and the network protocol it is important to characterize the dominant applications [4, 8, 9, 12, 19, 22, 23]. Only by utilizing data about all events initiated by the Web (including TCP and HTTP events) can one hope to understand the chain of performance problems that current Web users face. Due the the popularity of the Web it is crucial to understand how usage relates to the performance of the network, the servers, and the clients. Such comprehensive information is only available via packet monitoring. Unfortunately, extracting HTTP information from packet sniffer data is non-trivial due to the huge volume of data, the line speed of the monitored links, the need for continuous monitoring, and the need to preserve privacy. These needs translate into requirements for online processing and online extraction of the relevant data, the topic of this paper. The software described in this paper runs on the PacketScope monitor developed by AT&T Labs[1]. The PacketScope is deployed at several different locations within AT&T WorldNet, a production IP network, and AT&T Labs-Research. One PacketScope monitors T3 backbone links, another PacketScope may monitor traffic generated by a large set of modems on a FDDI ring or traffic on other FDDI rings, another PacketScope monitors traffic between AT&T Labs-Research and the Internet. First deployed in Spring 1997, the software has run without interruption for weeks at a time collecting and reconstructing detailed logs of millions of Web downloads with less than a worst case of 0.3% packet loss. The rest of this paper is organized as follows. Section 2 discusses the advantages of packet sniffing and Section 3 outlines some of the difficulties of extracting HTTP data from packet traces. The overall software architecture is described in Section 4. Our solution is presented in Section 5 and finally Section 6 briefly summarizes some of the lessons learned.